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Program Description^ 



Odyssey Math, published by CompassLearning®, is a web-based 
K-8 mathematics curricuium and assessment tool designed to 
aliow for instructionai differentiation and data-driven decision 
making. 2 The online program inciudes electronic curriculum and 
materials for individual or small group work, assessments aligned 
with state curriculum standards, and a data management system 
that allows teachers to develop individualized instructional and 
assessment tools, as well as track individual and classroom student 



performance. Odyssey Math can be used as a standalone curric- 
ulum or as a supplement to other mathematics curriculum. The 
primary school version of the Odyssey Math curriculum focuses on 
fundamental math skills like numeracy for the earlier grades, while 
in later grades, the curriculum equips students for skills necessary 
in middle and high school mathematics. The interactive activities 
used for both age groups allow for the application of ideas, tools, 
and manipulatives, and build upon previous knowledge. 



Ressarch^ No studies of Odyssey Math meet What Works Clearinghouse 
(WWC) evidence standards and one study meets WWC 
evidence standards with reservations. The study included 207 
fifth-grade students from one school district in Pennsylvania 



where Odyssey Math was used as a supplemental curriculum to 
Houghton Mifflin."^ 

Based on this study, the WWC considers the extent of evi- 
dence for Odyssey Math to be small for math achievement. 



EffactivanaSS Odyssey Math was found to have potentially positive effects on math achievement. 



Math achievement 

Rating of effectiveness Potentially positive 

Improvement index® +17 percentile points 

1. The descriptive information for this program was obtained from publiciy-avaiiabie sources: the program’s website (http://www.compassiearning.com, 
downioaded October 2008) and Brandt et ai. (2006). The WWC requests deveiopers to review the program description sections for accuracy from their 
perspective. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2. This review refers to studies of Odyssey Math in kindergarten through fifth grade. Studies of Odyssey Math conducted in sixth through eighth grade 
were out of the scope of the Eiementary Schooi Math protocoi. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 1.0 (see the WWC Standards). 

4. The evidence presented in this report is based on avaiiable research. Findings and conciusions may change as new research becomes avaiiabie. 

5. This number shows the student-ievei improvement index based on the findings in the one study that meets WWC evidence standards with reservations. 
That one study examined the effectiveness of Odyssey Math when it is used as a supplemental program. 
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Additional program Developer and contact 

information Odyssey Math was developed and is distributed by CompassLearn- 
ing, Inc. Address: 203 Colorado Street, Austin, TX 78701. Email: 
lnfo@compasslearnlng.com. Web: http://www.compasslearnlng.com. 
Telephone: (800) 232-9556. 

Scope of use 

CompassLearning was founded in 1969 and develops education 
software. The web-based Odyssey Math product was launched In 
2003 as a follow-up product to the original server-based Odyssey 
product (now termed Odyssey Classic) that was delivered under 
the Josten’s brand and then by CompassLearning. Odyssey Classic 
is no longer sold or supported by CompassLearning.® According 
to the developers, more than 11 million students in 20,000 schools 
nationwide use Odyssey Math and other Odyssey programs. 



Math can be used as a standalone curriculum or as a supple- 
ment to other mathematics curricula to provide an alternative 
instructional model, differentiated instruction, or gain feedback on 
particular concepts students are struggling with. Specifically, the 
management tool allows teachers to customize instruction for indi- 
vidual students and/or for classroom lesson plans. The program’s 
reporting system automatically prescribes individualized learning 
paths with activities that are tailored for each student based on his 
or her test scores. Teachers can also generate reports detailing 
student achievement and areas that need further review. Odyssey 
Math’s assessment tool provides a means to build and customize 
assessments based on key focus areas, individual learning styles, 
or local, state, and national standards. Teachers can use the bank 
of test items, or they can build their own items and item banks for 
any grade level and any subject. 



Teaching Cost 

Odyssey Math offers a full mathematics curriculum for grades K-8 Odyssey Math pricing depends on the Implementation, 
that combines both online and offline tools and activities. Odyssey 



Research Fourteen studies reviewed by the WWC investigated the effects 
of Odyssey Math. One study (DiLeo, 2007) is a randomized 
controlled trial that meets WWC evidence standards with 
reservations. The remaining 13 studies do not meet either WWC 
eligibility screens or evidence standards. 

DiLeo (2007) analyzed data from a randomized controlled 
trial (RCT) conducted in a single school district in Pennsylvania. 
Teachers and students were assigned to classrooms by princi- 
pals, with efforts made to balance classrooms by student ability. 
Teachers were then randomly assigned to use either Odyssey 
Language Arts or Odyssey Math in their classroom. The group 
using Odyssey Language Arts served as the control group to 



the Odyssey Math treatment group. Teachers were asked to 
use Odyssey Math at least 90 minutes per week (the developer- 
recommended amount) as a supplement to their core mathematics 
curriculum (Houghton Mifflin). The study focused primarily on 
the relationship between Odyssey usage (in the treatment group) 
and student achievement, but the author also examined outcome 
differences of the treatment and control groups. The original 
randomly assigned sample included 13 classrooms (7 treatment 
and 6 control) in five schools. A single magnet school with two 
comparison classrooms was dropped from the analysis because 
the school’s demographic composition was significantly different 
from the other schools in the sample. Therefore, the final analysis 



6. This review focuses only on studies of the newer, web-based Odyssey Math product launched in 2003. 
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Research (continued) 



sample included 11 classrooms (7 treatment and 4 control) in 
4 schools with a total of 207 fifth-grade students/ Despite the 
severe differential attrition of treatment and control classrooms, 
post-attrition treatment and control groups had only small 
differences on baseline test scores that were not statistically 
significant, but that did require the author control for the baseline 
pretest (according to the Elementary Math protocol).® Because 
of the differential attrition, the WWC rated this RCT as meeting 
evidence standards with reservations. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 
Standards Handbook, Appendix G). The extent of evidence 
takes into account the number of studies and the total sample 
size across the studies that meet WWC evidence standards with 
or without reservations.® 

The WWC considers the extent of evidence for Odyssey Math 
to be small for math achievement. 



Effectiveness 



Findings 

The WWC review of interventions for Elementary School Math 
addresses student outcomes in math achievement. The findings 
below present the author’s estimates and WWC-calculated 
estimates of the size and the statistical significance of the effects 
of Odyssey Math on students.''® 

DiLeo (2007) reported a statistically significant positive effect 
on mathematics achievement as measured by the Pennsylvania 
System of School Assessment (PSSA) between the students 
randomly assigned to use Odyssey Math (the “treatment” group) 
and Odyssey Reading (the “control” group). In WWC calculations, 
this effect was not statistically significant. However, the WWC 
determined the effects were large enough to be considered 



substantively important (that is, an effect size of at least 0.25). 
Therefore, based on this one study, Odyssey Math appears to 
have potentially positive effects on mathematics achievement. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research design, 
the statistical significance of the findings, the size of the difference 
between participants in the intervention and the comparison condi- 
tions, and the consistency in findings across studies (see the WWC 
Procedures and Standards Handbook, Appendix E). 



7. An additional school with 73 students was Included In the author’s analysis but Is not Included in this report because teachers in that school were not 
randomly assigned to use either the Language Arts or Math software (as described below) and could have used both, and therefore were classified as a 
separate group by the author. The author provided the WWC with sample sizes and standard deviations, details about the random assignment process, 
and Information about attrition not presented In the published 2007 study, which was a dissertation. 

8. The author presented third-grade Pennsylvania System of School Assessment (PSSA) scores to establish baseline equivalence. Fourth-grade students 
did not take the PSSA In 2005, so third grade was the most recent year available. 

9. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the Intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings In which studies took place— are not taken Into account for the categorization. Information about how the extent of evidence rating was 
determined for Odyssey Math Is In Appendix A5. 

10. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate 
the statistical significance, see WWC Procedures and Standards Flandbook, Appendix C for clustering and WWC Procedures and Standards Flandbook, 
Appendix D for multiple comparisons. In the case of DiLeo (2007), a correction for clustering was needed, so the significance levels may differ from 
those reported In the original study. 
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The WWC found 
Odyssey Math to have 
potentially positive 
effects for math 
achievement 



References 



Improvement index 

The WWC computes an improvement index for each individuai 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Procedures 
and Standards Handbook, Appendix F). The improvement index 
represents the difference between the percentile rank of the aver- 
age student in the intervention condition versus the percentile rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is based entireiy 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analyses. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting results favorable to the intervention group. 



Based on the one study of Odyssey Math that meets evidence 
standards with reservations, the improvement index for math 
achievement is +17 percentile points. 

Summary 

The WWC reviewed 14 studies of Odyssey Math. One of these 
studies meets WWC evidence standards with reservations. 

The remaining 13 studies do not meet either WWC evidence 
standards or eligibility screens. Based on the one study, the 
WWC found potentially positive effects in math achievement. 

The conclusions presented in this report may change as new 
research emerges. 
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the measures of effect cannot be attributed solely to the 
intervention— there was only one unit of analysis in one or 
both conditions. 

CompassLearning, Inc. (2007). Impact of CompassLearning 
Odyssey Reading/Language Arts & Mathematics on NWEA 
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Appendix 



Appendix A1 Study Characteristics: DiLeo, 2007 



Characteristic 


Description 


Study citation 


DiLeo, J. (2007). A study of a specific ianguage arts and mathematics software program: Is there a correlation between usage levels and achievement? Unpublished doctoral 
dissertation, Indiana University of Pennsylvania, Indiana, PA. 


Participants^ 


In five ouf of fen schools wifhin a single school district, fifth-grade teachers were randomly assigned to use CompassLearning’s Odyssey Language Arts or Odyssey Math in their 
classrooms. Random assignment of the thirteen study teachers occurred after students were assigned to classrooms by their principals. The students in the Odyssey Language Arts 
classrooms served as the control group to the students using Odyssey Math (the intervention of interest for this WWC review). Two classrooms of students from one school were 
removed from the analysis sample because the school was a magnet school that had different demographic composition from the other schools. The final analysis sample, after 
excluding children with missing data, included 4 schools with 7 treatment classrooms (125 students) and 4 comparison classrooms (82 students). The author presented third-grade 
Pennsylvania System of School Assessment (PSSA) scores to establish baseline equivalence. Fourth-grade students did not take the PSSA in 2005, so third grade was the most 
recent year available. Post-attrition treatment and control groups had small differences on baseline test scores that were not statistically significant, but that did require the author 
control the baseline pretest (according to the Elementary Math protocol). Because the differential attrition rates between the treatment and comparison groups at the classroom 
level were greater than 5%, but the author provided the appropriate demonstration of equivalence and a control for the pretest, the WWC rated this study as meeting evidence 
standards with reservations. Approximately 7 percent of the students in the analysis sample were non-white, 63 percent of the students qualified for free or reduced-price lunch, 
and 14 percent of the students had an Individualized Education Program (lEP). An additional school was also presented in the author’s analysis but is not included in this report 
because teachers in that school were not randomly assigned to use either the Language Arts or Math software and could have used both. 


Setting 


The study was conducted in one school district in central Pennsylvania. 


Intervention 


The intervention condition consisted of using Odyssey Maf/r during the 2005-06 school year in addition to the usual mathematics curriculum, Houghton Mifflin. Teachers were 
asked to use the software a minimum of 90 minutes per week (the developer-recommended minimum), but usage levels varied across classrooms, at least in part because 
of access to technology. In two of the schools, students could only access the software during their weekly assigned time in the computer labs. In the other two schools, 
students had greater access to the software, as it was available during their weekly computer labs, in their classrooms via wireless laptops, and even at home. Students in 
the Odyssey Math condition could use the Odyssey software for any subject except Language Arts. Some of the students (from two of the schools) had access to the Odyssey 
software in the 4th grade — the year before the study began. 


Comparison 


The students in the control condition used only the district’s Houghton Mifflin curriculum for mathematics. These students used the Or/yssey software for Language Arts and 
possibly for other subjects (other than Mathematics). Some of the students (from two of the schools) had access to the Odyssey software in the 4th grade — the year before 
the study began. 


Primary outcomes 
and measurement 


Math achievement was measured using the Pennsylvania System of School Assessment (PSSA), which is the standardized assessment used for state accountability. Eor a 
more detailed description of this outcome measure, see Appendix A2. 


Staff/teacher training 


It is not clear how much training the teachers in this study received. However, the district had substantial funds to purchase professional development from CompassLearning 
for teachers in two of the schools during the 2004-05 school year through an Enhancing Education Through Technology (EETT) grant from the Pennsylvania Department 
of Education. During the 2005-06 school year, teachers who used Odyssey during the previous (2004-05) school year trained teachers using it for the first time in 2005-06. 



1. Sample sizes and information about attrition and random assignment were provided to the WWC by the study author. 
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Appendix A2 Outcome measures for the math achievement domain 



Outcome measure 


Description 


Pennsylvania System 
of School Assessment 
(PSSA)i 


According to the Pennsylvania State Department of Education, the annual PSSA is a standards-based criterion-referenced assessment. The PSSA assesses students’ abilities 
relative to specific standards within each subject and for each grade level. There are specific cut-off scores to determine a student’s proficiency level. Students receive 
designations of below basic, basic, proficient, or advanced, depending on how they score in each tested subject. In addition to achievement categories, a continuous measure 
of student achievement also is presented in the study. The continuous score is presented in this intervention report. 



1. A description of the Pennsyivania System of Schooi Assessment was obtained at the Pennsyivania State Department of Education’s website (http://www.pde.state.pa.us/a_and_t/site/default.asp) 
in December 2008. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


Odyssey Math Comparison 

group group 


Mean 

difference^ 
(Odyssey Math- 
comparisonj 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 







DiLeo, 2007 ^ 










Pennsylvania System of School Grades 11/207 


1583 


1480 


103 


0,45 


ns 


+17 


Assessment 


(239) 


(215) 










Domain average for math achievement (DiLeo, 2007)^ 








0.45 


ns 


+17 



ns = not statistically significant 



1. This appendix reports the findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The author provided the WWC with standard deviations and 
means adjusted for pretest differences. Pretest adjusted means are presented for the Odyssey Math and comparison groups. 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of DiLeo (2007), a correction for clustering 
was needed, so the significance levels may differ from those reported in the original study. 

8. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. 

The domain improvement index is calculated from the average effect size. 



WWC Intervention Report Odyssey Math 



August 2009 



8 





Appendix A4 Odyssey Math rating for the math achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of math achievement, the WWC rated Odyssey Math as potentiaiiy positive. It could not achieve a rating of positive because there was 
oniy one study. The remaining ratings (mixed, no discernibie effects, potentially negative, or negative) were not considered, as Odyssey Math was assigned the highest 
appiicabie rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect. 

Met. Odyssey Math had one study that showed a substantively important positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. Odyssey Math had no studies showing negative effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Odyssey Math had no studies with a strong design. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. Odyssey Math had no studies showing negative effects. 



1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5 Extent of evidence 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Math achievement 


1 


4 




207 


Small 



1. A rating of “medium to iarge” requires at ieast two studies and two schoois across studies in one domain and a totai sampie size across studies of at ieast 350 students or 14 ciassrooms. 
Otherwise, the rating is “smaii.” For more detaiis on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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